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evaluation report and an evaluator"s recommendations X^s examined* 
Sabjects read one of four evaluatioa reports about testing and 
grading procedurea in a achool prfogramp The reports varied in the 
amount of jargon and data used to justify the recommenaations* 
Ninety-five high school teachers and administratora read one of four 
ahort reports, each containing one of the following types of 
statementai (1) Jargon-Loaded, Objective, (2) Jargon-Pree, Ob jective, 
{3} Jargon^Loaded, Subjectige, and (4) Jargon^Pree, Subjective* She 
Jargon^Loaded reports Here rated aa more technical than the 
Jargon-Free reports. The leaat difficult format was the Jargon-Free 
Subjective report and the moat difficult waa the Jargon-Loaded 
Subjective report. The Subjective reporta were rated aa more 
practical and the Jargon-Loaded Subjective reporta ii^.re rated as less 
believable than Jargon**Free Objective reports^, There wr^re no 
differences in reactions to the recommendationa of the evaluator, Th 
results suggest that the impact of an evaluation report depends upon 
the style in Mhicji it is written* (Author) 
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Inlmm Credibility mi Mcapwcci as a Functioii 

of Saport Sfiyli: 

Do Jsr|on and Oats Ml i Dif firines!^ 

hkn D, Brom 
The MviEiiEy of Nilraika-linwk 

Larry A, Briikarap 
The University of Illlnpii it Urbani-Chaf il|n 

Diinna L, Nma 
Ihs Univarsity of NibWika-lincoln 

fhi ivikatioii of coipensitory and innovativi prs|riii in iduMtion, , 

itrvicis in ffliatil mi wnunlty hialthj mi ioclital riforie has iriitid long 

iiiough to havt pstigne of Its utility end worth ralaad by ill typis of publici, 

IflcMlng avikitori themiilvai. Do policy mi diclaion Mkire utllizi tviliii- 

tivi Inforsation and, furthaiiori, ire thiy Influineid by thi fiidbiek thiy . 

reciivel 

Mkm riprtini thi lipaet of ivaluatlfln on decliion-i^ln| is Ditly 
ixpifientiil aid oftin iuggiits th|t ths influincHa not as powirfyl ai iliht be ;. 
hppid (Gubaj 1569; Wortta h Mm, 1973; ftrgln, Wi), Davis and Silailn (1515) 
in tliilr extlttsivi mim of thi utilization of ivilyitlon itatij ^'A portion of thi 
apparent defflprflHiation iioni vitirin ivilaitors My be ittrlbutid to the ilgw 
prociss of utlUiation of evalmtion risult^J' (p, i25) Howivi);, they alio point 
out that thi utilization rati is battir whin the ifflpli;t of ivaluitlon on jiciilon 
liking ii aseeased over an iKtandid period of tlnii, 

Siveril iugptloni for iiiereasini thi utiliEation of iviluativi InforMtlon 
in policy and decllion ilkin| have hm dnmi Davii and Siliiin (1W5) su||iit 
■ that iviluatori trik'.thi uii of thiir ivaluatlons to datiet Am and hm the riiylts 
are usidi Stiki (1??5) and Houii {Wl)^ amoni othare^ have luipitid that a uieful 
tvaluatioD ii oni that ii reippniiva to thi inforMl liiidi of decialon mkth . 
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Fomil rsporti of m ivaluition |eneriUy coiitiin jar|aa which provide pri- 
ciif and efflciint wmnication to coliiiiuis and indudi ixtfiniive reftrincis to 
dita to livi sbjactivt iuppdrt to the evaluator'i ritomnditiDni. laforiation snd " 
cQmicition thioryj howivtrj ^joiild luipit a pessibli Intswetlve iffict iepisdini ■ 
upon thi type of audlincii Carter (19?1) and Briikaipj Brown and iiiin (1916), for 
i^Kaiple, have repdrtad that lOMptanei of eviluation riports Is ralitid to a coiplix 
lit of viriibliij inckdlnj the title and deierlptlon of the evakatorj thi niture 
of thg proirii bilni iviluatid, and thi iilf^intiriite of thi ludiinci* Others in 
coMunicitloai laiiiroh have found that the uie of inforMtion will be greater if 
thi ludlinci reprds the praiintition as coherent and tindaritatidabla (Davie and 

Thii study Mined the iipsct of dlffirint evaluation report stylei on audlenei 
agriiaint with an ivaluator'i recoiiindationij auiienci rating of the evaluator'i 
crtdlWllty, ^d ludlenQe asseiisiant of the ivaluition report, Two diraanslons of 
report itylis ^iira etudiidi (i) theirant of profisiioi|al jirgonj and (b) the usi 
of dita4iiid it^tiienti. The pitioni of interiit iire whither the use of jarp 
inhanci or detriet froi iudience icciptanee of thi recoiianditioni and froi their 
ratinp of the utility of thi evaluation reports, and shithir dati-basid or lubjictlvi- 
biiid ititiisinti in thi iViluation report affect aadienci ratings of the objectivity 
and biliiviblUty of the iviluitor, 
Mithodoloiy 

Subjects were lakid to read i iiEiulatii evaluatlnn report that was bitten in 
four different ways and to give thiir riactloni to thi ivaluator and the ivaluation 
. report, Ihi sontixt of the iviliiitlon riport wai Gownity Goncirn ibout tisting 
and grading in the public ichocl!, A cltigene' eoyacil asked an sxttrnal eviluator 
to pther wUvint dati-^ld laki ippfopEiite rieoMiitions to thii, Thi ivilui" 
tlon riport wae in idvocacy report focualng on four lieuiii (1) Oii of critirion* 
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referenced standards In assessing student classroom perfor^nce, (2) Need for more 
parent conferences^ (3) Assessment of student effort. as well as achievement^ and 
(6) Assessment of non-^academic accomplishments. Each reconmendation advocated 
changes in current practices followed by a brief justificatioti. Subjects were 
asked to read the recommendation and its justification before indicating their 

4 . ■ - .... 

own viewpoints. 

Four reports (treatments) were developed ^hich had the same issues, recommen- 
dations, lengthj concepts J and ar^ments. Each justification varied, however^ in 
the amount of jargon and/or data-based statements. The four treatments werei 
(1) Jargon-Loaded Objective^ (2) Jargon-Loaded Subjective ^ (3) Jargon-Free Objectlv 
and (4) Jargon-Free Subjective. Jargon words ^re operationally defined as thoae 
which succinctly conveyed a concept to a professional audience, which were in fre-* 
quent usage In an educational setting^ and for which there were more general usage 
words or phrases available. These included such words as "norm-referenced," "com- 
petency," "transpersdnal and "psychomotor*'* Objective-based statements were 
operationally defined as statements which reported local and national survey 
results and cited percentages. Subjective-based statements included phrases like 
"1 believe, "I think. "In my opinion, , ." to emphasize the evaluator*3 
personal opinions. 

Subjects included 95 teachers and publie school administrators who were 
Involved in a variety vorkshops and self-improvement courses at the University 
of Nebraska-Lincoln, These subjects were assumed to be familiar with educational 
jargon and had some cKperlence with evaluation reports* As educators they had 
undoubtedly thought through Issues related to grading, W'hich was the focus of the 
report* , 

Dependent measures Included assessment of: (1) Agreement with the evaluator^s 
recommendations I (2) Ratings of the evaluator on thorougtaesi, self-confidence. 



knowiedgft about testing and grading, believabillty , awareness of school needs ^ 
logic, practicality 3 convincing and objectivity | and (3) Batings of the avaluaClon 
report on amount of technical terms^and difficulty level. 

After each issue the subjects indlcsted their agreement on a five point 
scale (Strongly Agree to Strongly Disagrett) to four specific recommendations 
advocated by the evaluator* The extent of subject agreement with the evaluator's 
four rGCQnmiendatlDns for each of the four issues was used as the measure of agree- 
ment* At the end of the simulation each subject rated the evaluator and the report 
by indicating hip responses on a five point scale (Vety High t:o Very Low) to eleven 
items measuring the dependent variables listed above* 
R esults 

Xhe respondents' extent of agreement with the evaluator-s recommendations 
^QXki analyzed using a two-factor analysis of variance* Table 1 presents the 
suHOTary of tte analysis of variance and Table 2 presents the means and standard 
deviations. No scatistically significant interactions or main effects, use of 
Jargon and use of data were obtained. 



Insert T^vble 1 and 2 about here 



Responses to the nine ratings of the evaluator and the two items assessing 
t-hc svaluaticn repcrt were analysed using a two-factor muitivarlate analysis of 
variance. The summaiy of the F-s are prosented in Table 3 and the means and 
standard deviations of all eleven items are presented in Table 4* 



Insert Table 3 and 4 abouc here 
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There was no significant multivariate interaction when the 11 rating 
items were analysed as a set* The mulcivariate F for the Data factor was not 
siinif leant. 

The multivariate F for the jargon factor was significant at the *05 
level* EKamination of the size of the univariate F's indicate that the major 
contributions to the overall effect of the use of jargon were on ratings of 
difficulty s technicality 5 knowledge^ and practicality* EKamination of the 
meriis in Table 4 indicate that the Jargon-loaded report was perceived as more 
difficult and more technical^ but the evaluator was perceived as more know- 
ledgeable and practical* The evaluator aa author of the jargon-leaded objec- 
tive report received the highest ratings on thoroughness, knowledge, believa- 
bilityj and practicality* The jargon-loaded oubjecclve report was viewed as 
the most technical and difficult of the four reports. 

There was a significant 'univariate F for interaction on ratings of 
difficulty of the report. EKamination of the means in Table 4 indicates that 
this interaction was disordinal with the jargon-loaded subjective report being 
perceived as most difficult and the Jargon-free subjective being perceived as 
least difficult* 
Discugsion and Iroplications 

These results suggest that the impact of an evaluation report depends 
upon the style in which it is written. The use n£ technical educational Jargon 
in particular J appears to have an impact upon nhe receptivity of an audience 
of educators* The jargon-»loaded reports were clearly perceived as more techni- 
cal by the readers than the jargon-free reports | buC^ interestingly ^ the 
jargon-loaded subjective report was perceived as more technical than the 
Jargon- loaded objective. Similarly, the teachers and administrators perceived 
the reports with jargon-loaded statements to be more difficult but their 
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perception was liifluenced by the Inclusion of data-based statements* The 
most difficult report was the Jargon-loaded subjective followed by the jargon- 
loaded objective, jargon-free objective, and the least difficulty the jargon- 
free subjective. It may be that the use of data^ In this case simple percen- 
tages, gave the reader B^aningful anchor points for Interpretation* 

Use of data to support evaluator viewpoints did not by Itself contri- 
bute Co higher ratings of the evaljator* However, when data and jargon were 
combined, the impact was apparent. The writer of the jargon- loaded reports 
was rated as more practical and knowledgeable than the jargon-free author. 
Though there was no significant Interaction, exaiftlnation of the means In 
Table 4 indicates that the major contribution to the overall difference came 
from the higher ratings given to the jargon-loaded objective reports* Data 
support did help make the evaluator appear more knowledgeable and the report 
more practical. The same pattern, though not significant, was true for 
ratings of thoroughness and believabllity* In each instance the Jargon^ 
loaded objective report was rated highest. 

The lack of any slgnificanc difference or pattern on ratings of '*con- 
vlncing'V is congruent with the lack of glgnlf leant differences In extent of 
agreement with the evaluator *fi recomfflendatlona * There are a number of possible 
eKplanatlons for this finding. The issues dealt with In the repcfrt* grading 
and testing, are ones for which most educators have rather definite opinions. 
Educators may be receptive to change , but not In the evaluation format used 
in this study. The opinions of the educators were strong enough to resist 
being influenced, despite the fact that the rsport style variations did affect 
their ratings of the evaluator and the usefulnesi of the report. 

The results of this study confirm the eKpectatlon that evaluation report 
styles do affect audience perceptldnSg Far educators the use of jargon and data 
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results In high ratings on soma dimensions, such as practicality and know- 
ledge j but does not seem to influence their acceptance of the evaluator's 
recoTOiendatlons on concroversial Issues, Further Inveetigation is needed on 
the effects of differing report styles for a variety of audience types on 
different Issuei* How would members of the P.T,A* react to the same reports? 
How would educatora react to slTnilar report variations on different topics? 



9 



References 

Braskamp, L,A* , Brown, R.D, ^ and Newman^ D.L, Evaluator credibility as a 

fmqtlon of the avaluator^s professional background and the client's status 
within the organizatipn ^ Paper presented annual meeting of American 
Psychological Association^ Washington,, D*C.j September 1976, 

Carter^ R*K, Client reactions to negative findings and the latent conserva- 
tive function of evaluation studies* The American SocioloRlst a 1971, 6^ 
118-124. 

Davis, H,R. and Salasin^ S*E. The utilii^atlon of evaluation. In EtL* Struening 
Mid M. Guttentagj (Eds,) Handbook of evaluation research ; Vol, 1, Beverly 
Hlllsi Sage Publications, 1975, pp. 621-666. 

Guba^ E.G, The failure of educational evaluation* Educational Technology » 
1969, £, 29-38. 

Housep E.R, The conscience of evaluation. Teachers C ollege Record ^ 1973, 
_3, 405-44. 

Stake, R. (Ed.). Evaluating the arts in educatloni A resparjSive approach. 

Columbus, Ohio: Charles A. Merrill, 1975, 
Wergin, J.F, The evaluation of organisational policy making i A political 

modal. Review of Educational Research , 1976, 4£, 75-116. 
Worthan, B, and Sanders, J. Educational evaluation! Theory and practice . 

Worthington, Ohlos Charles Jones, 1973. 



10 



Jargott, Page 9 

Tabla 1 

toalyais of Variance Summary on EKtent of Agreement tflth 
Evaluator*9 Recomendatlons 



Source df MS P 



Jargon (A) 




1 


32.08 .65 




Data Support (B) 




1 


17.63 .35 




Interaction (AxB) 




1 


6.08 .12 




Within 




91 


49.33 




Total 




95 


105.12 










Table 2 


„ ^ _ 


Means 


and 


Standard DevlatlonB on Extent of Agreement with 








Evaluator' 


8 Recommendations 








Jargon-Loaded 


Jargon-Free 




Objective 


X 


57.33 


59.04 






SD 


7.32 


6.44 






N 


24 


22 




Subjective 


X 


58.66 


59.36 





SD 8.09 5.72 
N 27 22 
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Table 3 



Sumnary of Multivariate Analyels of Ratings of 
Evaluator and Evaluation Report 



Source of Variation Multivariate F Univariate F 



Interaction .92 
Scale : 

Thoroughness 1,13 

Salf-ConfidMce t38 

Kaowladge 3.48 

Beliavability 1.40 

Logical .00 

Pr^ictieal 1.96 

Awareness 1*99 

Convincing .09 

Objectivity .17 

Technicality 1.45 

Difficulty S.64* 

Data Support ,92 

Scale: 

Thoroughness 1,02 

Self-Confidence .33 

Khowladge tSZ 

Believabillty ,84 

Logical ■ ,22 

Practical .17 

Awareness . .61 

Convincing ' .00 

^bjnativlty 1*72 

Techniealtty 1.89 

Difficulty .01 
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Scale: 2.66* 

Thoroughne IS *40 

Self -Confidence . .70 

Knowledge 2,63 

Believability *02 

Logical .41 

f actlcal 2.18 

Awareness .00 

Convincing . #12 

Objectivity .32 

Technicality 23.70* 

Difficulty:. ' 12.26* 



*Sipiif leant at ,03 level 



Means and Standatd Deviations of Ratings of Evaluator 
and Evaluation Report 

Jargon-Loaded Jargon^Loadad Jaifg©n«F«a Jargon-Free 

Objective - Subjectiva Objective Subjectivii 

N^27 N=22 Ni-22 





X 


SD 


X 


SD 




SD 


X 


SD 


Thoreughness 


3.79 


.59 


3.46 


.88 


3.50 


.96 


3.52 


.73 


Self Ceafldence 


4.00 


.51 


4.00 


.90 


3.73 


1.08 


3.96 


.96 


about easting 
and grading 


4.25 


.74 


3.89 


.68 


3.73 


.63 


3.01 


.7i 


Bellevablllty 


3,88 


.80 


3.54 


92 


3.64 


.79 


3.70 


.76 


Logical 


3.71 


1.00 


3.78 


.79 


3.82 


1.09 


3.01 


.67 


Fractlcallty 


3,67 


1.09 


3.46 


1.23 


3.00 


1.31 


3.43 


.73 


jtorarenees of School 
Needs 


3.58 


1.28 


3.46 


1.04 


3.27 


1.24 


3.78 


.74 


Convlnelng 


3.58 


1.39 


3.54 


.92 


3.59 


.73 


3.65 


.71 


Objectivity 


3.71 


.86 


3.39 


1.13 


3.73 


.83 


3.56 


.73 


Technicality 


3.33 


1.09 


3.82 


.90 


2.64 


1.09 


2.65 


.71 


Difficulty 


2.71 


1.04 


3.14 


.88 


2.50 


1.10 
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